Method, apparatus and software for providing user feedback

ABSTRACT

A device for providing feedback to a user of a string instrument is described. The device comprises receiving circuitry operable in use to communicate with at least one finger sensor capable of detecting the force of a finger on a string, a camera operable in use to capture an image and a display operable in use to display data, wherein the receiving circuitry, camera and display are connected to processing circuitry which is operable to: detect, from the receiving circuitry, the force of a user&#39;s finger on at least one string at a given time; detect, from the camera, the position of the user&#39;s finger on the at least one string at the given time; compare the detected force of the user&#39;s finger and the position of the user&#39;s finger on the at least one string at the given time with a predetermined value of force and position at the given time; and generate, on the basis of the said comparison, feedback to display on the display.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to United Kingdom Application1313391.3 filed on 26, Jul. 2013 the contents of which beingincorporated herein by reference in its entirety.

BACKGROUND

1. Field of the Disclosure

The present disclosure relates to a method, apparatus and system forlearning to play a musical instrument.

2. Description of the Related Art

The “background” description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thebackground section, as well as aspects of the description which may nototherwise qualify as prior art at the time of filing, are neitherexpressly or impliedly admitted as prior art against the presentdisclosure.

Learning to play a musical instrument has historically taken placeeither by a player learning via traditional music materials such asmusic books or by being taught by someone who already knows how to play(or, indeed, a combination of these methods). The problem with thesemethods, however, is that music books offer no interaction with theuser, and thus the user is required simply to persevere with anyproblems that they encounter. Also, if a user wishes to learn fromsomebody else, they will generally have to pay for tuition, which can bevery costly.

In recent years, however, with the development of computers and theinternet, it has been possible for people to learn to play a musicalinstrument by following virtual lessons. These virtual lessons can bebought on disc (such as a CD-ROM or DVD-ROM) or can be downloaded orstreamed from the internet (from YouTube®, for example). These virtuallessons can include videos and interactive software applications to helpthe player learn.

The problem, however, is that the virtual lessons available today tendto constitute only one-sided tuition. The user is able to learn from andpossibly interact with the video/software application. However, theyreceive no direct feedback on the way in which they are performing withthe instrument. The present disclosure helps to address these issues.

SUMMARY

In the disclosure there is described a method of providing feedback to auser of a string instrument, comprising the steps: detecting the forceof a user's finger on at least one string at a given time; detecting theposition of the user's finger on the at least one string at the giventime; comparing the detected force of the user's finger and the positionof the user's finger on the at least one string at the given time with apredetermined value of force and position at the given time; andgenerating feedback to display to the user on the basis of the saidcomparison.

The foregoing paragraphs have been provided by way of generalintroduction, and are not intended to limit the scope of the followingclaims. The described embodiments, together with further advantages,will be best understood by reference to the following detaileddescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 shows a system according to an embodiment of the presentdisclosure;

FIG. 2 shows a finger tip sensor according to an embodiment of thepresent disclosure;

FIG. 3 shows a 3-axis tactile sensor unit according to an embodiment ofthe present disclosure;

FIG. 4 shows a tablet computer on which embodiments of the presentdisclosure can be implemented;

FIG. 5 schematically illustrates the inputs and outputs a correlationfunction engine according to an embodiment of the present disclosure;

FIG. 6A shows an example of a correlation function CF according to anembodiment of the present disclosure;

FIG. 6B shows a fret/string calibration procedure according to anembodiment of the present disclosure;

FIG. 6C shows a method for calculating the vertical and horizontalangles of frets/strings according to an embodiment of the presentdisclosure;

FIG. 6D shows a live image of a user playing the guitar whilst wearing aplurality of finger tip sensors;

FIG. 6E shows a process in which the true finger positions of the userare mapped for a sample of collected image data ID, force data FD and/oraudio data AD according to embodiments of the present disclosure;

FIG. 7 shows a first correlation function comparison method according toembodiments of the present disclosure;

FIG. 8 shows a second correlation function comparison method accordingto embodiments of the present disclosure;

FIG. 9A shows a tablet computer showing a song selection screenaccording to embodiments of the present disclosure;

FIG. 9B shows a tablet computer showing a set-up screen according toembodiments of the present disclosure;

FIG. 9C shows a tablet computer showing a performance screen accordingto embodiments of the present disclosure;

FIG. 9D shows a tablet computer showing a results screen according toembodiments of the present disclosure;

FIG. 10 shows a process by which user correlation function data isobtained and compared with professional correlation data and by whichuser feedback is generated, according to embodiments;

FIG. 11 shows a system over which users may compete in competitions,according to embodiments;

FIG. 12 shows a tablet computer showing an online competition screenaccording to embodiments of the present disclosure; and

FIG. 13 shows a process by which a user may create or join onlinecompetitions, according to embodiments.

DESCRIPTION OF THE EMBODIMENTS

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts throughout the several views.

FIG. 1 shows a system according to an embodiment of the presentdisclosure. Included in the system are a tablet computer 100(hereinafter referred to as a tablet) and a plurality of finger tipsensors 102A-D which can each be worn on a finger 104 of a user. Each ofthe finger tip sensors 102A-D are a different colour. As will beexplained later, the tablet 100 is operable to run software whichprovides musical instrument tuition to the user. The tablet is able toreceive information transmitted from the finger tip sensors 102A-102Dwhen they are worn by the user as the user is playing a musicalinstrument, and the software is then able to provide feedback to theuser about how they can improve their performance.

FIG. 2 shows a finger tip sensor 102 according to an embodiment of thedisclosure. The finger tip sensor includes a finger tip casing 200 intowhich a finger of the user can be inserted and a 3-axis tactile sensorunit 202. The finger tip casing can be constructed from any suitablematerial which is comfortable for the user to wear and which does notinhibit the user's ability to play a musical instrument. For example,the casing 200 could be made from silicone rubber. As will be explainedlater, the 3-axis tactile sensor unit is for detecting the force appliedby the user by the finger on which the finger tip sensor 102 is worn asthe instrument is played, and for transmitting this force information tothe tablet 100. A suitable finger sensor is described in a paper by M.Hori et. al “3-Axis Fingertip Force During Playing the StringInstrument”, Transducers 2013, Barcelona, Spain 16-20 Jun. 2013. Thecontents of this paper is incorporated in its entirety by reference.

In embodiments, the instrument played by the user is a stringinstrument, such as guitar. Thus, the force measured by the 3-axistactile sensor unit will be the force exerted on a string of the guitarby the user as the user pushes the string to make contact with the fretboard to play a note with the guitar. It will, however, be appreciatedthat any instrument for which the user makes notes by pressing a part ofthe instrument with their finger could be used with embodiments of thepresent disclosure.

FIG. 3 shows an embodiment of the 3-axis tactile sensor unit 202 in moredetail. It can be seen that the sensor unit comprises a 3-axis tactilesensor 300, a transmitter 302 and a control unit 304. The 3-axis tactilesensor is able to measure the force exerted on it (and thus the forceexerted by a finger of the user when the finger tip sensor 102 is wornby the user) in 3 dimensions. The transmitter 302 is then able totransmit data of the force and the frequency of that force measured bythe 3-axis tactile sensor to the tablet 100. The transmitted 302 may beof any suitable wired or wireless type. For example, the transmitter 302may transmit the force data via a Wi-Fi, Bluetooth, radio frequency (RF)or infrared (IR) connection with the tablet 100. The control unit 304manages the transfer of data between the 3-axis tactile sensor 300 andthe transmitter 302.

FIG. 4 shows an embodiment of the tablet 100 in more detail. It can beseen that the tablet 100 comprises a central processing unit (CPU) 400,a memory 402 and a storage unit 414. The storage unit may comprise anysuitable medium for storing electronic data, such as a flash memory unitor hard disk drive (HDD). The software application for providing theinteractive musical tuition to the user is stored in the storage unit414 and is executed by the CPU 400 when an instruction is received fromthe user to run the software application.

The tablet 100 also comprises a microphone 404 for receiving audio data,a camera 406 for receiving image data and a receiver 408 for receivingthe force data transmitted by the transmitter 302 of the finger tipsensor 102. As will be discussed later, the musical tuition softwareapplication may use at least one of the audio data, image data and forcedata received by the tablet for providing feedback to the user on theirperformance with the musical instrument.

The tablet also comprises a display 412 for displaying information tothe user and a wireless data transceiver 410 for connecting to a network(such as the internet). The wireless data transceiver may, for example,be a Wi-Fi or wireless mobile network transceiver. It is noted that, inembodiments where the force data is transmitted from the finger tipsensor 102 to the tablet 100 over a wireless network such as a Wi-Finetwork, the separate receiver 408 in the tablet 100 may be omitted andthe force data may be received via the wireless data transceiver 410instead. In this case, the wireless data transceiver acts as thereceiver 408.

FIG. 5 schematically illustrates the inputs and outputs of the musicalinstrument tuition software application that is run by the tablet 100.The software application includes a correlation function engine 500which converts the audio data AD, force data FD and/or image data IDreceived by the tablet 100 during a musical instrument performance intoa correlation function CF represents the musical instrument performanceand allows different musical instrument performances of a particularsong to be compared with each other.

As will be explained later, a user is thus able to play a particularsong and have their performance recorded as a user correlation function.The user correlation function can then be compared with a professionalcorrelation function which has previously been recorded by aprofessional musician. By comparing the user and professionalcorrelation functions, the user can see the areas of the performance inwhich they need to improve, so that their performance more accuratelyreflects that of the professional musician. Thus, unlike traditionalelectronic musical tuition, the user is advantageously able to obtainpersonalised feedback on their actual musical performance.

FIG. 6A shows an example of a correlation function CF according to anembodiment of the present disclosure. The correlation function CFcomprises a table 600 recording various parameters of the musicalperformance which have been obtained from the audio data AD, force dataFD and/or image data ID received by the tablet 100. The table 600includes four columns 602, 604, 606 and, 608 and is a table whichrelates to a single finger tip sensor 102 worn by the user. A furthercolumn 620 in the table 600 is included that relates to the soundreceived from the microphone 404 of the tablet. A separate table 600will be generated for each finger tip sensor 102 worn by the user, andthe combination of all the tables gives the correlation function CF.

Column 602 shows the time elapsed during the performance. Samples of theaudio AD, force FD and/or image ID data will be captured at apredetermined sample rate, and the elapsed time during the performanceeach time a sample is captured will be recorded in column 602. In thisexample, a sample is captured at a rate of 100 samples per second,resulting in the elapsed performance time incrementing in 0.01 secondintervals. However, it will be appreciated that any other suitablesample rate could be used. Such sample rates could be 22.6 μS (or asampling frequency of 44.1 kHz which is the same as a regular compactdisc). Other sample frequencies include 40 kHz for Human Hearing, 32 kHzfor transmission-related application, or 48 kHz recommended by The AudioEngineering Society. A suitable sample rate should be such that thegranularity of the performance is maintained in the correlation functionCF.

Column 604 shows the force measured (in Newtons) and transmitted by thefinger tip sensor 102 at the time in column 602. The force is measuredand transmitted by the 3-axis tactile sensor unit 202 at thepredetermined sample rate and is a 3-vector indicating the force in thex, y and z directions. In this case, the z direction refers to thedirection in which the string of the guitar is pressed towards the fretboard so as to change length of the vibrating part of the string andhence change the musical note. The majority of the force exerted by theuser will thus be in the z direction. However, there may also usually bea smaller force in the x and y directions. These will be due to a shearforce exerted by the user on the string as the string is pressed down.

Column 606 shows the position of the finger of the user on which thefinger tip sensor 102 is worn. The position has two components, thesebeing the one or more strings of the guitar which are pressed down bythe user 610 and the number of the fret onto which the one or morestrings are pressed 612. For ease of explanation here, the one or morestrings pressed down are represented using a binary number, whereas thefret number (like all other numbers in table 600) is represented using adecimal number. It will be appreciated, however, that since this is asystem implemented electronically, that all numbers will, in reality, beprocessed by the CPU 400 of the tablet 100 as binary numbers.

It can be seen that, for the first captured sample at time 0.01 seconds,the string position is given as “100000” and the fret position is givenas “7”. This means that a single string (the bass string of the guitar)has been recorded as being pressed down at fret 7. On the other hand,for the tenth sample captured at time 0.10 seconds, the string positionis given as “011000” and the fret position is given as “6”. This meansthat two strings (the 2^(nd) and 3^(rd) strings from the base string ofthe guitar) have been recorded as being pressed down at fret 6.

The position of the finger on which the finger tip sensor 102 is worn isdetected from the image data ID captured by the camera 406 of the tablet100 using any suitable image analysis method which is able to determinethe position of the coloured finger tip sensor with respect to themarked frets of the guitar fret board and the sharp edges defined by theguitar strings. An example image analysis method is explained later withreference to FIGS. 6B-6D.

Column 608 shows a frequency spectrum 614 for each captured sample. Thisfrequency spectrum can be obtained from the rapid force changesdetectable by the 3-axis tactile sensor unit 202 as a string pressed bythe user's finger vibrates (assuming that a finger tip sensor is worn onthis finger), and by performing a Fourier transform.

Column 620 shows a frequency spectrum 622 for each captured sample. Thisfrequency spectrum is the frequency spectrum of the sound receivedthrough the microphone 404. The sound received through the microphone404 has a Fourier transform performed thereon. This may be used incombination with the frequency spectrum obtained in column 608.

In embodiments in which the frequency spectrum is obtained from therapid force changes measured and transmitted by the 3-axis tactilesensor unit 202, this can be advantageous, because it allows the numberof strings which are pressed down by the user to be easily determined.

This is possible because the harmonics of a single string, which arerepresented by peaks in the frequency spectrum, are multiples of thefundamental (first) harmonic. Thus, for a single string, the peaks inthe frequency spectrum will be evenly spaced with respect to thefrequency (f). However, for more than one string, there will be morethan one fundamental harmonic (one for each string) and a set ofassociated further harmonics for each of those fundamental harmonics.Each set of further harmonics will comprise multiples the respectivefundamental harmonic, thus resulting in peaks which are not evenlyspaced but, rather, have a spacing which alternates between two or morevalues (depending on the number of strings pressed). The frequencyspectrum captured at a particular sample can thus be analysed by thecorrelation engine 500 using any suitable method that more than onestring has been pressed by the user.

It is noted that each of audio data AD, force data FD and/or image dataID captured by the tablet 100 can be used in combination in order toensure that the table 600 is as accurate as possible.

For example, it has been discussed that the image data ID captured bythe tablet 100 can be analysed so as to determine the position of acoloured finger tip sensor 102 worn by a finger of the user as they playa musical instrument such as a guitar. In this case, however, afterperforming the image analysis, the correlation function engine 500 maybe unsure as to whether the user is actually pressing down a string orsimply has their finger positioned above the string so that it appears,from the image data, that the string is being pressed down. In thiscase, the correlation function engine 500 will cross check with theforce data for the sample in order to make a decision. So, for example,if the force in the z direction (F_(Z)) is below a certain thresholdvalue, then it will be determined that the user is not pressing down thestring. On the other hand, if F_(z) is above the certain thresholdvalue, then it will be determined that the user is pressing down on thestring.

Multiple thresholds could also be introduced. So, for example, if F_(Z)is below a first, lower threshold, then it is determined that the useris not pressing down on the string. On the other hand, if F_(Z) is abovea second, higher threshold, then it is determined that the user is notpressing down on the string. However, if F_(Z) is between the first,lower threshold and the second, higher threshold, then it is determinedthat the user's finger is merely resting on the string, so as to producea soft, harmonic sound with the guitar.

As another example, the position of a user's finger can be initiallydetermined using the image data ID captured by the camera 406 in thetablet 100. However, if multiple strings have been pressed by the user'sfinger (as occurs, for example, when the user attempts a “barredchord”), then this may be difficult to detect purely from the imagedata. For example, it may be difficult to differentiate between whetherthe user is pressing down multiple strings, or whether the user isactually just pressing down a single string and has their fingerstretched across the other strings (stretching across other strings isnecessary, for example, when pressing down any string other than thehighest pitched string of a guitar).

Thus, for a particular sample, the image data can be cross-referencedwith the frequency spectrum in order to determine whether or not morethan one string has been pressed down. More specifically, if only asingle string has been pressed down, then there will be only one set ofpeaks in the frequency spectrum, with each peak being separated by aconstant frequency difference. On the other hand, if more than onestring has been pressed down, then there will be, accordingly, more thanone set of peaks and more than one frequency difference between peaks.

FIGS. 6B-6D show, in more detail, how the correlation functionexemplified by the table 600 is created for each finger sensor 102A-D.

As explained with reference to FIG. 6A, the correlation function engine500 is able to use image data from the camera 406 of the tablet 600 todetect the positions of the finger sensors as the user plays a guitar.The finger sensor positions are determined with respect to the stringsand frets of the guitar.

In order to facilitate accurate detection of the finger sensorpositions, it must be ensured that the correlation function engine 500can accurately determine the positions of the strings and frets of theguitar. Optionally, in embodiments, the string and fret positions areset during a calibration procedure by the user. The calibrationprocedure is carried out using a fret/string calibration screen 617 onthe tablet 100, as shown in FIG. 6B.

On the calibration screen 617, there is shown a live image 615 of theuser's guitar which is being captured by the camera 406 of the tablet100. The user has positioned the tablet and guitar so that the fretboard 620 of the guitar, including the first fret 621, can be seen bythe camera.

In order for the correlation function engine 500 to determine the firstfret of the guitar, a marker 618 has been positioned on the guitar bythe user above the nut 623 of the guitar (the nut 623 defining the endof the fret board). This marker may be provided to the user at the sametime as the purchase of the finger tip sensors, and includes apredetermined colouring and/or pattern which is easily detectable by thecorrelation function engine 500. The marker may be releasable fixed tothe guitar using any suitable method.

Once the marker has been detected in the image 615, the correlationfunction engine 500 then performs an edge detection process on the image615. The edge detection process can take place using any suitablemethod, such as Canny edge detection, Marr-Hildreth edge detection or aHough transform. During the edge detection process, the sharp andwell-defined appearance of the strings 624 and the fret bars 626 meansthat the strings and fret bars are detected as edges.

The correlation function engine 500 then decides which of the detectededges correspond to fret bars and which correspond to strings. Thisdecision process can occur using any suitable method. For example, thecorrelation function engine 500 can decide that detected edges which aresubstantially parallel to a reference edge defined by the marker 618(the marker 618 can be seen in the image 615 as extending away from thefret board 620, defining a reference edge) correspond to fret bars andthat detected edges which are substantially perpendicular to thereference edge defined by the marker 618 correspond to strings.

The vertical and horizontal angles of each detected edge (including thereference edge) can be determined using the method shown in FIG. 6C,which shows a close-up of the fret board 620. Here, the detected edgesof the guitar strings 624 and a fret bar 626 can be seen. An example ofthe angle calculation is shown for the edge defined by the fret bar 626.

In FIG. 6C, the ends of the fret bar edge 626 have been determined as(x₁, y₁) and (x₂, y₂). The horizontal (φ_(H)) and vertical (φ_(V)) canthen be determined by the following formulae:

$\varphi_{H} = {\tan^{- 1}\frac{\left( {{y_{1} - y_{2}}} \right)}{\left( {{x_{1} - x_{2}}} \right)}}$$\varphi_{V} = {\tan^{- 1}\frac{\left( {{x_{1} - x_{2}}} \right)}{\left( {{y_{1} - y_{2}}} \right)}}$

Thus, for each of the detected edges, the horizontal (φ_(H)) and/orvertical (φ_(V)) angles for each edge can be calculated and comparedwith the corresponding angles of the reference edge of the marker 618.Edges which have the same the horizontal (φ_(H)) and/or vertical (φ_(V))angles as the reference edge (to within a reasonable degree of error,such as 0.5°) are determined as parallel to the reference edge and thusdetermined as corresponding to fret bars. On the other hand, edges whichhave horizontal (φ_(H)) and/or vertical (φ_(V)) angles which differ by90° to those of the reference edge (again, to within a reasonable degreeof accuracy, such as 0.5°) are determined as perpendicular to thereference edge and thus determined as corresponding to strings. Allother detected edges in the scene (that is, those which are determinedas being neither parallel nor perpendicular to the reference edge, andwhich thus correspond to neither a fret bar nor a string) are discarded.

The detected fret bars 626 and strings 624 and then numbered by thecorrelation function engine 500. The fret bars 626 are consecutivelynumbered in ascending order, starting from the fret bar closest to themarker 618. So, the fret bar closest to the marker is labelled 1, thenext fret bar is labelled 2, the next 3, etc. The fret bar numberingwill continue until the last fret bar that can be seen and detected inthe image has been numbered. Similarly, the strings are consecutivelynumbered in ascending order, starting from the string closest to the topof the image (which corresponds to the lowest bass string of theguitar). So, the bass string is labelled 1, the next string is labelled2, etc. The string numbering will continue until all six strings of theguitar have been numbered.

The number associated with each fret bar is overlaid on top of the liveimage 615 in bubbles 628. Similarly, the number associated with eachstring is overlaid on top of the live image 615 in bubbles 630. Thisallows the user to inspect the automatic fret bar and string numberingthat has occurred during the calibration procedure.

As a final check, the user may manually correct any fret bar or stringedges which have been incorrectly detected and numbered by thecalibration procedure. Such an incorrect detection and numbering mayoccur if, for example, the image 615 captured by the camera 406comprises a large number of edges which do not correspond to either afret bar or string but which have been nonetheless been detected as suchby the calibration procedure. The user may perform the correction bymanipulating the number bubbles 628, 630 using the touch screenfunctionality of the tablet 100.

For example, if an extra edge has been accidently determined as a fretbar (this could occur if there was an extra edge in the background ofthe image 615 which did not correspond to a fret bar, but which happenedto be parallel to the reference edge of the marker), then the user mayselect the number bubble 628 associated with that edge and drag it withtheir finger to the discard region 619 of the calibration screen 617.This would signal to the correlation function engine 500 that the edgeassociated with the dragged number bubble 628 is not actually a fretbar, and that the edge should thus be discarded. In this case, thewrongly detected edge is discarded and, if necessary, the numbering ofthe remaining fret bars is amended (this is necessary if the wronglydetected edge appears between two genuine fret bar edges, in which casethe deletion of the wrongly detected edge will result in a missingconsecutive number if the numbering is not amended).

Once the automatic calibration process and any manual corrections havebeen completed, the user presses the continue button 625 to continue tothe guitar lesson.

Once the strings 624 and fret bars 626 have been detected, the user maythen start playing the guitar and the correlation function engine 500may record the positions of the user's fingers with respect to thestrings and fret bars. This is illustrated in FIG. 6D.

FIG. 6D shows a live image 615 of the user playing the guitar whilstwearing the finger tip sensors 102A-D, with one finger tip sensor oneach finger. For simplicity, the numbering of the strings and fret barsis not shown. However, the correlation function engine 500 will be awareof the string and fret bar numbering, and will use this numbering todetermine the position of each of the finger sensors. The finger sensors102A-D are each a different colour so that each finger sensor can beindividually followed by the correlation function engine 500 in the liveimage 615 using any suitable colour/hue detection algorithm. Inembodiments, the colours may be chosen to be particularly bright orunusual (such as bright primary colours) so that the finger tip sensorscan be easily detected in the image using a combination of edgedetection (such as Canny edge detection, Marr-Hildreth edge detection ora Hough transform) and colour/hue detection. Various colour/huedetection algorithms are well known in the art and thus will not beconsidered in detail here.

Through the detection and numbering of the strings 625 and fret bars 626and the detection of the coloured finger tip sensors 102A-D, thecorrelation function engine 500 is able to determine the position ofeach of the finger tip sensors with respect to a string and fret bar.

For the fret bar position, the detected fret bar immediately to the leftof the finger tip sensor in the image 615 (and thus to the right of thefinger tip sensor in real life) is determined as the relevant fret bar.This ensures that the determined fret bar number corresponds to the“fret number” (the frets actually being the regions between adjacentfret bars) that is recorded in table 600.

For the string position, then the uppermost string (that is, the stringcloset to the top of the image 615) with which the finger tip sensorappears to intersect is determined as the relevant string. If multiplestrings have been pressed, then, as will be explained later, this isdetermined with respect to the frequency spectrum generated from theforce data detected by the sensor.

The strings and fret bars thus effectively define a grid upon which thepositions of the finger sensors 102A-D can be defined. Using the methoddescribed above (and with reference to the fret bar and string numberingin FIG. 6B), finger tip sensor 102A is thus determined to be positionedat fret 5 and string 3. Similarly, finger tip sensor 102B is determinedto be positioned at fret 5 and string 3, finger tip sensor 102C isdetermined to be positioned at fret 4 and string 4, and finger tipsensor 102D is determined to be positioned at fret 4 and string 1.

It is noted that, on certain occasions, particular fret bars 626 maybecome temporarily hidden between consecutively captured images as theuser moves their hand and fingers around the fret board. This problemcan be alleviated by the correlation function engine 500 interpolatingthe position of a fret bar 626 which cannot be seen in a particularimage 615 using data from previous and/or subsequent images. Thecorrelation function engine 500 will know when to include aninterpolated fret bar by measuring the distance between adjacent fretbars during the calibration procedure illustrated in FIG. 6B and thenmeasuring the distance between adjacent visible fret bars for eachcaptured image 615. When the distance between adjacent visible fret barsin a captured image is found to exceed the distance between those fretbars found during the calibration procedure (allowing for an appropriatemargin of error, such as 10% or 20%, so as to account for the fact thatthe user may move the guitar closer to the camera 406 and that theobserved distance between adjacent fret bars may thus legitimatelyappear to increase), then an error is recorded and an interpolated fretbar is added between the two visible adjacent fret bars.

In embodiments in which the distance between visible adjacent fret barsis measured for each captured image 615 and during the calibrationprocedure, the measured distances will be the distances from the marker618 to fret bar 1, from fret bar 2 to fret bar 3, from fret bar 3 tofret bar 4, etc. This means that, for the captured images 615 (aftercalibration), if an error is found, for example, between visible fretmarks 2 and 3, then an interpolated fret bar is added between fret marks2 and 3 and the numbering of the fret bars is changed to reflect this.So, fret bar 2 remains as fret bar 2, fret bar 3 becomes fret bar 4, andthe new, interpolated fret bar becomes fret bar 3. Also, all fret barswith numbering subsequent to original fret bar 3 have their numberingincreased by one. This ensures that the numbering of the fret bars iscorrected so as to take into account the fact that an interpolated fretbar has been added.

FIG. 6E shows a process 631 in which the true finger positions of theuser are mapped for each sample of collected image data ID, force dataFD and/or audio data AD according to embodiments of the presentdisclosure.

The process starts at step 632. At step 634, the sample of image data,force data and audio data is captured by the tablet 100. As alreadymentioned, the image data ID is captured using the camera 406 of thetablet 100, the force data is generated by the finger tip sensors 102A-Dand transmitted to the tablet 100 and the audio data AD is capturedusing the microphone 404 of the tablet 100.

At step 636, the image data in the form of the captured image 615 isanalysed according to the methods previously described so as todetermined, for a first finger tip sensor worn by the user, the positionof the finger tip sensor with respect to a fret number and a stringnumber. At this stage, no analysis has been carried out as to whether ornot the user is pressing down more than one string with a single finger,and thus the determined finger tip sensor position is only an initialposition which may be amended later in the process 631.

At step 638, it is then determined, from the force data generated by thefinger tip sensor, whether the average force applied by the user'sfinger over the sample collection period is above a first threshold.This first threshold is a predetermined threshold above which it isassumed that the user is pressing fully down on the string(s) and so asto shorten the string and change its pitch.

It is noted that the sample collection period mentioned here refers tothe period of time over which the sample is collected. For example, thesample collection period may be the time period which corresponds to thesample rate (so, for example, a sample rate of 100 samples per secondhas a corresponding time period of 0.01 seconds, which would also be thesample collection period). Alternatively, the sample collection periodmay be smaller, so as to reduce the amount of processing required foreach sample (so, for example, a sample rate of 100 samples per secondcould have a corresponding sample collection period of 0.005 seconds,resulting in a reduced quantity of data to analyse before the nextsample is taken).

If it is determined that the average force is above the first threshold,then the process moves on to step 640. Here, the rapidly-varying forcedata generated by the finger tip sensor (due to the vibration of thepushed down string(s)) over the sample period is analysed so as toproduce a frequency spectrum. The frequency spectrum is produced byperforming a Fourier transform on the rapidly-varying force data.

On the other hand, if it is determined that the average force is belowthe first threshold, then the process moves on to step 642, where it isdetermined whether the average force is above a second threshold, thesecond threshold being lower than the first. The second threshold is apredetermined threshold above which it is assumed that the user is onlylightly touching the string so as to produce a soft, harmonic sound withthe guitar.

If it is determined that the average force is above the secondthreshold, then the process moves to step 644, where the finger tipsensor position is recorded as being related to a harmonic. The processthen moves onto step 640, where a Fourier transform is applied to therapidly-varying force data generated by the finger tip sensor.

On the other hand, if it is determined that the average force is belowthe second threshold, then it is assumed that no strings are beingtouched by the finger tip sensor (the string is being played openly),and thus that the initial position of the finger tip sensor asdetermined from the image data in step 636 actually related only to theuser's finger being above the strings without touching them. The processthus moves on to step 646 in which the initial position of the finger isupdated to record the fact that no string has been pressed. In thiscase, the finger position can be recorded as (000000, 0) in table 600.The process then moves on to step 648, in which it is determined as towhether or not there are any remaining figure tip sensors 102A-D whichhave not yet been analysed.

In the case that the average force generated by the finger tip sensor isdetermined to be above the first or second threshold, the process movesfrom step 640 to step 650, in which it is determined, from the Fouriertransform of the rapidly-varying force data, whether or not there ismore than one set of harmonic peaks. As already mentioned, in the casethat only a single string has been pressed, there will be only a singleset of peaks, with each of the peaks separated by a constant frequencydifference. On the other hand, if more than one string has been pressed,then harmonics from each of the pressed strings will be detected,resulting in more than one set of peaks (one set for each stringpressed). In this case, the frequency difference between the peaks willvary between several values, with the total number of differentfrequency difference values being indicative of the number of pressedstrings.

In the case that there is not more than one set of peaks, the processmoves on to step 652. Here, the initial position of the finger tipsensor, in which the finger tip sensor is positioned at only one string,the one string being the uppermost string (that is, the string closet tothe top of the image 615) with which the finger tip sensor appears tointersect, is confirmed.

On the other hand, in the case that there is more than one set of peaks,the process moves on to step 654. Here, the initial finger sensor tipposition is amended so as to include a number of additional, adjacentstrings. The number of additional strings is determined according to thenumber of additional sets of peaks (and thus the number of additionalfrequency difference values) detected in the Fourier transform. So, forexample, if two sets of peaks were detected, then this means that thereis one additional set of peaks (compared to the pressing down of asingle string), and thus one additional, adjacent string is added to theinitial finger position. Alternatively, if three sets of peaks weredetected, then this means that there are two additional sets of peaks(compared to the pressing down of a single string), and thus twoadditional, adjacent strings are added to the initial finger position.

The adjacent strings added are the next consecutively numbered stringswith respect to the string recorded in the initial finger tip sensorposition. So, for example, if the initial finger tip sensor positionrecords string 2 (see FIG. 6B) as being pressed down (corresponding toan initial finger tip sensor string position of 010000), then oneadditional string will result in the addition of string 3 (correspondingto a new finger tip sensor string position of 011000). Alternatively,two additional strings will result in the addition of strings 3 and 4(corresponding to a new finger tip sensor string position of 011100).

The above method works on the assumption that a user can only press downadjacent strings with a single finger (rather than non-adjacentstrings). Advantageously, by using this method, in which image data iscomplemented with force data from the finger tip sensors, the fingerpositioning of a user can be accurately determined without having toperform advanced image processing techniques for determining whichstrings have been pressed and which have not. Rather, the image dataonly needs to be analysed to determine the uppermost string pressed downby the user and the remaining information regarding additional strings,harmonics, etc. is obtained from the force data. The reduction of a needfor advanced image processing techniques reduces the amount ofprocessing required in obtaining the finger position of the user.

After the initial finger tip sensor position has been updated, theprocess moves on to step 648. Here, it is determined whether or notthere are remaining finger tip sensors 102A-D which have not yet beenanalysed for the sample. If there are remaining finger tip sensors, thenthe process moves to step 636, at which point the finger positiondetermination process is repeated. On the other hand, if all finger tipsensors have been analysed for the sample, then the process moves ontostep 656.

At step 656. it is determined whether or not the sample time period haselapsed. The sample time period is the corresponding time period to thesample rate (so, for example, a sample rate of 100 samples per secondwould have a sample time period of 0.01 seconds). If the sample timeperiod has not yet elapsed, then step 656 is repeated. On the otherhand, if the sample time period has elapsed, then the process returns tostep 634, at which point the next sample of image data ID, force data FDand audio data AD is taken.

It is noted that, throughout the process 631 of FIG. 6E, the Fouriertransform and frequency analysis of the strings is conducted solelythrough the force data provided by the finger tip sensors 102A-D.However, in embodiments, it is also envisaged that the audio data ADpicked up by the microphone could also be used to analyse thefrequencies of the strings and to determine which strings have beenplucked. This analysis could be based on, for example, particularcharacteristics of the strings which are known to the correlationfunction engine 500 in advance. Alternatively, the audio data AD couldsimply be recorded, and associated with the each of the samples in thecorrelation function, so that a user can record a performance with theirmusical instrument and, based on a certain part of their performance asdetermined by the audio recording, look at the correlation functionassociated with that certain part and obtain feedback (the way in whichfeedback may be presented to the user is described later on).

Once the user correlation function has been obtained for a user'sperformance, then this correlation function can be compared to aprofessional correlation function that has been obtained for aperformance by a professional musician. It should be noted that thecorrelation function for the professional musician will be captured inthe same manner as for the user explained above. There are many ways inwhich the user and professional correlation functions could be compared.

A first comparison method, according to embodiments, is shown in FIG. 7.Here, for each captured sample of the user and professional correlationfunctions (captured samples with the same captured time corresponding tothe same point in a musical performance for the user and professional),the user finger tip sensor position 702 is compared with theprofessional finger tip sensor position 704. An error value 706 can thenbe determined from this comparison.

For a first sample (at 0.01 seconds), both the string and fret positionsof the user match that of the professional. There is therefore no error,and the error value 706 is recorded as 0.

For a second sample (at 0.10 seconds), the string position of the usermatches that of the professional. However, the fret position of the userdoes not match, meaning that the user has made an error with the fretposition. Due to the fact that a single error has occurred (fretposition only), the error value 706 is recorded as 1.

For a third sample (at 0.25 seconds), neither the string position northe fret position of the user matches that of the professional, meaningthat the user has made an error both with the string and fret positions.Due to a fact that a double error has occurred (string and fretposition), the error value 706 is recorded as 2.

The sample comparisons and error values shown for a single finger tipsensor in FIG. 7 will be generated for each of the user's finger tipsensors. As will be seen, this will then allow feedback to be given tothe user as to which parts of the performance they need to improve sothat their performance more closely matches that of the professional.

It will be appreciated that any form of error value system could beestablished following the comparison of the user and professional fingerpositions. Here, a simple two-tier error system has been introduced (anerror value of 1 for a single fret or string error and an error value of2 for both a fret and string error). However, there are many othersystems which could be implemented. For example, the error couldincrease depending on how many strings are incorrect (so that a singlebit discrepancy in the comparative string positions of the user andprofessional would result in an error value of 1, two bit discrepancieswould result in an error value of 2, etc.) or how large the fret erroris (so that a fret position of 6 instead of 5 would result in an errorof 1, a fret position of 7 instead of 5 would result in an error of 2,etc.).

The error value system could also take into account additional factorsto the positions recorded in the table of FIG. 7, such as whether ornot, for a certain string and fret position, the force exerted on thestring by the user is above the first force threshold (indicating thatthe string has been fully pressed down) or is between the first andsecond force thresholds (indicating that the string is merely beinglightly touched so as to produce a soft, harmonic sound with theguitar). This could be indicated by an additional harmonic bit, forexample, with 0 indicating a normal pressing of the string and 1indicating a lighter press of the string. The user's harmonic bit couldthen be compared with the professional's harmonic bit, and an errorvalue 706 of 1 could be generated in the case that the user andprofessional harmonics bits do not match for a certain sample.

A second comparison method, according to embodiments, is shown in FIG.8. Here, a graph 800 of the average force in the z direction (that is,towards the fret board of the guitar) over the sample collection periodfor each captured sample is shown against time for a single finger tipsensor. The dashed line 802 shows the average force for the user and thesolid line 804 shows the average force for the professional. Thediscrepancy between the average force of the user and the average forceof the professional at a given time t_(n) is given by ΔF_(zn).

The pressure exerted by a player on the strings of a guitar can affectthe sound of the guitar. Thus, in order for the user's performance tomatch the professional's performance as closely as possible, the usermay wish to exert a force on the strings with each finger which issimilar to that used by the professional. Thus, by use of the graph 800,the user may identify time periods during their performance in which theforce they exert closely matches that of the professional (for example,time period T₂). They may also identify time periods during theirperformance in which the force they exert is significantly different tothat of the professional (for example, time period T₁), and where theythus may need to improve.

As well as identifying particular regions of the performance which mayrequire improvement with regards to the force applied to the strings, anoverall score of the force difference between the user and theprofessional may be calculated using the force difference ΔF_(zn) foreach of the captured samples at time t_(n). This overall score may begiven as:

$\begin{matrix}{{{Total}\mspace{14mu}{Error}} = \sqrt{\sum\limits_{n}\left( {\Delta\; F_{Zn}} \right)^{2}}} & (1)\end{matrix}$

The user may thus work to reduce the total error in their performancewith regards to the force exerted on the strings. A graph 800 like thatshown in FIG. 8 may be generated for each of the user's finger tipsensors and the total errors for each of the finger tip sensors may beadded to give an overall error which the user can try and reduce astheir musical performance is improved.

FIGS. 9A-9D show a process by which, according to embodiments, a userselects a song to perform and performs the song so that the performancecan be converted to a correlation function CF using the correlationfunction engine 500. The user interface described here may be interactedwith by the user via a touch sensitive screen of the tablet 100.

In FIG. 9A, after an appropriate piece of application software (“anapp”) is downloaded to the tablet and opened on the tablet, a songselection screen 900 is shown on the tablet. The tablet may be providedwith the app pre-installed at purchase or may be downloaded from anappropriate app store either free of charge or for a nominal sum ofmoney. The finger sensors described above may be provided free of chargewith the tablet, or may be provided to the user after download of theapp. The song selection screen shows a selection of songs 902 which theuser can select. These will be songs which have been purchasedelectronically by the user and downloaded from an external server (notshown) or may be provided free of charge on the tablet or with apurchased album. Included in the downloaded song will be a professionalcorrelation function which has been previously recorded by aprofessional musician, together with audio and/or video data of theprofessional musician's performance. As described above, by including aprofessional correlation function with the song, a correlation functiongenerated by the user can be compared with this professional correlationfunction so that the user can obtain feedback on how their performancemay be improved.

Each of the songs 902 are identified by an artist 904, a track name 906and a difficulty rating 908. The difficulty rating gives an indicationof how difficult the song may be to perform, so that a user can choose asong which is appropriate to the musical skills and experience. Thedifficulty rating will be decided in advance and provided as metadatawith the professional correlation function and audio and/or video datawhich defines the song. On the song selection screen 900, the user alsohas the option of pressing the virtual button 910, which will direct theuser to an online store in which more songs can be purchased.

The user is able to select a song to perform simply by touching theappropriate song 902 on the song selection screen 900. Once a song hasbeen selected, the screen changes to the set-up screen 912, asillustrated in FIG. 9B. It can be seen that here, the user has chosenthe song “Here Comes the Sun” by The Beatles.

The set-up screen 912 allows the user to tune their guitar so that itmatches the tuning of the guitar used by the professional musician whenthe professional performance was recorded and the professionalcorrelation function was generated. This ensures that the comparison ofany audio data AD for the user and professional performances ismeaningful, in that corresponding strings of the user and professionalguitars generate the same frequencies. Also, if a backing track isprovided for the user by the tablet 100 as the user records theirmusical performance, then it is necessary that the user's guitar istuned so that it is in the same key as the backing track.

During the guitar set-up, the user works their way through each stringof the guitar, starting from the string 1 with the lowest natural pitch(the lowest bass string of the guitar) and ending with string 6 with thehighest natural pitch (the highest pitch string of the guitar). Thenumbering of the strings is indicated graphically by number bubbles 926,and matches the string numbering used in the previously describedcorrelation function generating process.

In order to tune each string, the user plucks the string openly (thatis, without pressing the string down at any point on the fret board),the microphone 404 of the tablet picks up the sound and the frequenciesof the string are analysed and compared with those of the correspondingprofessional string frequencies. The tuning adjustment icon 916 thenindicates whether or not a tuning adjustment is required and what thistuning adjustment should be. Methods of comparing the frequencies of aplucked string for the purpose of guitar tuning are well known in theart and are thus not described in detail here.

In the example in FIG. 9B, the tuning adjustment icon 916 is an upwardspointing arrow, indicating that the pitch of the string needs to beincreased and thus that the string should be tightened. The user maythus tighten the string using the tuning keys of the guitar (not shown).In the case that the pitch of the string needs to be reduced, the tuningadjustment icon 916 may instead by a downwards pointing arrow. Thisindicates to the user that the tightness of the string needs to bereduced, again by turning the tuning keys of the guitar (although in theopposite direction). In the case that the pitch of the string iscorrect, then the tuning adjustment icon 916 can indicate this in anysuitable way. For example, the tuning adjustment icon 916 could read“OK” or similar.

As the user adjusts the tension of the string in accordance with theupwards and/or downwards arrows of the tuning adjustment icon 916, thepitch of the string will eventually match the pitch of the correspondingprofessionally tuned guitar string. At this point, the tuning adjustmenticon 916 will indicate that the pitch of the string is correct (byreading “OK” or similar) and the user may then press the virtual button918 to progress to the next string.

The progress of the string tuning process is indicated by graphicalstrings 914 and the associated tuning indicators 920, 922 and 924. Forstrings which have been correctly tuned (strings 1, 2 and 3), the tuningindicator is a tick 924. For the string which is currently being tuned(string 4), the tuning indicator is an arrow 922. Graphics may also beapplied to the string which is currently being tuned. For example, thegraphical string may be made to vibrate when the microphone of thetablet 100 detects that the string has been plucked. For strings whichare still yet to be tuned (strings 5 and 6), the tuning indicator is adot 920. Of course, the string indicators may comprise any appropriategraphics which indicate to the user the tuning status of each string.

Once all strings have been correctly tuned (that is, after string 6 hasbeen correctly tuned), the user will press the virtual button 918 oncemore to start the recording of their musical performance. During theperformance, a performance screen 928 is displayed to the user. This isillustrated in FIG. 9C.

The performance screen 928 shows a real time live video image 930 of theuser playing their guitar which has been captured by the camera 406. Inthe captured image, the strings 624, fret bars 626, marker 618 andfinger tip sensors 102A-D can be seen. As described earlier, processingis performed on the live image 930 and on force data FD generated by thefinger tip sensors 102A-D and/or audio data AD detected by themicrophone 404 so as to detect the positions of the user's fingers withrespect to the strings 624 and frets 626 of the guitar. This processingis performed at a predetermined sample rate and the finger positions ofthe user at each sample are recorded so that they may be compared withthe corresponding finger positions of a professional musician.

In embodiments, an audio recording of the professional performance ofthe song is provided to the user as a backing track by way of speakers(not shown) included in the tablet 100. The provision of theprofessional performance as a backing track is highly advantageous, asit allows the user to aurally match their performance with that of theprofessional (perhaps with an additional rhythm, vocals, bass line,etc.) and to quickly determine which part of the song they are at duringa performance. The professional performance used for the audio backingtrack will be the same performance for which a professional correlationfunction CF has been recorded.

The time elapsed during the performance is indicated by the progress bar932 and progress indicator 933. An actual time (in minutes and seconds,for example), could also be included on the performance screen 928(although this is not shown). The user may touch and drag the progressindicator so as to move between different parts of the songs. Thisallows the user to practise a particular part of the song which theyfeel they need to improve, for example. As the user moves the progressindicator 933, the backing track is skipped to the part of the songindicated by the progress indicator 933, thus allowing the user torepeat a desired section of the performance with the backing track.

As well as the progress bar 932 and progress indicator 933, there isalso provided a song section indicator 934 which indicates predeterminedsections 939 of the song which a user is likely to want to repeat. Forexample, the song may be divided into various verses, a chorus and abridge, and the user may want to be able to quickly skip to one of thesesections 939 without having to manually find the section using theprogress indicator.

The song section which is currently being played may be indicatedgraphically using any suitable method, such as highlighting 937. Theuser may select a particular song section 939 simply by touching it. Atthis point, the backing track will skip to the beginning of the chosensection and the user's correlation function data (that is, the imagedata ID, force data FD and/or audio data AD which define the user'scorrelation function) will be recorded from a time in the songcorresponding to the beginning of that chosen section.

It is noted that as the user skips between different parts of the song,either by way of the progress bar 932 and progress indicator 933 or byway of the song section indicator 934, the backing track and the time atwhich the user's correlation function starts being sampled and recordedis adjusted according to the elapsed song time at the chosen part of thesong.

So, for example, if the user skips to time 1:36:23 in the song bytouching and dragging the progress indicator 933, then the backing trackwill be skipped to time 1:36:23 and the correlation function data of theuser will be sampled and recorded from time 1:36:23 (so, for example, anew record will be added to table 600 in FIG. 6A at the predeterminedsample rate from time 1:36:23). Similarly, if the user skips to thefirst chorus, which happens to be at time 0:48:20, then the backingtrack will be skipped to time 0:48:20 and the correlation function dataof the user will be sampled and recorded from time 0:48:20.

It is noted that if correlation function data has already been recordedby the correlation function engine 500 for a particular time which theuser has skipped to (this could occur if, for example, the user decidesto go back to a particular part of the song which they have alreadyperformed but which they wish to improve), then the previous correlationfunction data is overwritten. Advantageously, this allows a user to goback and improve a particular part of the song that they have alreadyplayed, thus improving the extent to which that part of theirperformance matches that of the professional. The user can do thiswithout having to repeat the entire performance again.

The performance screen 928 also includes a speed bar 936 and speedindicator 935 which allows the user to adjust the speed of the backingtrack and thus the speed at which they are expected to play so as tomatch their performance with that of the professional. If the usertouches and drags the speed indicator 635 in an upwards direction, thenthe speed of the backing track is increased and the correlation functiondata samples are obtained from the user's performance at a faster rate.On the other hand, if the user touches and drags the speed indicator 935in a downwards direction, then the speed of the backing track is reducedand the correlation function data samples are obtained from the user'sperformance at a reduced rate. Advantageously, this allows the user toadjust the speed at which they are expected to play during theperformance. Inexperienced users can thus choose to play more slowly,whereas more experienced users can play at an increased speed. Inembodiments, the maximum speed may be the speed at which theprofessional musician made the original recording.

It is noted that as the speed at which the backing track is played andat which the user function correlation data is sampled is adjusted, thetime elapsed during the performance remains unchanged as a marker of theextent to which the performance has been completed. Thus, for example,if the performance is recorded as being 3:12:59 minutes long, then thisdoes not change in the case that the performance is slowed down, eventhough, in reality, it may take more than a time of 3:12:59 for the userto complete the performance with the backing track. Similarly, thesample rate is judged with respect to the unchanged marker. So, forexample, if, at the full speed performance, a sample is captured every0.01 seconds, then samples will continue to be captured every time theelapsed time changes by 0:0:01, even though, in reality, the sample ratewill be reduced for a slowed down track. Thus, for a track which isplayed at half speed, a sample will be captured every time the elapsedtime changes by 0:0:01, but, in reality, this will actually equate to asample being captured every 0.02 seconds.

Advantageously, this removes the need to adjust the elapsed time andpredetermined sample rate as the speed of the performance is changed,thus reducing the amount of processing required by the correlationfunction engine 500.

During the recording of a performance, the user may press the virtualbutton 940 to toggle between the live video image of their ownperformance 930 and a pre-recorded video image of the professionalmusician (now shown) at a corresponding point in the performance (asdetermined by the elapsed time in the performance). The video image ofthe professional musician will be the corresponding video data to theaudio data of the backing track. Advantageously, this allows the user tovisually compare their performance with that of the professional in realtime. The video image of the professional musician can be skipped usingthe progress bar 932 or song section indicator 934 and slowed down/spedup using the speed bar 936 in exactly the same way as previouslydescribed.

There may also be a further toggle screen accessible by pressing thevirtual button 940 in which the live video image of the user'sperformance 930 and pre-recorded video image of the professional'sperformance are placed side-by-side in a split-screen arrangement (nowshown). Advantageously, in this case, the user is able to directlyvisually compare their own performance (such as their fingerpositioning) with that of the professional musician whilst looking in asingle place (that is, at the screen of the tablet 100). In this case,through successive presses of the virtual button 940, the video shown onthe performance screen 928 will toggle between the live video image ofthe user performance, the pre-recorded video image of the professionalperformance and the split-screen arrangement.

The performance screen 928 can also include an advice box 938 whichoffers real time advice to the user on how to improve their performance.This is based on a real time comparison of the user correlation functionand the professional correlation function. The generation of real timeadvice is explained later on.

After the performance has finished, a results screen 942 is displayed tothe user. This presents a range of different types of feedbackinformation based on a comparison between the newly generated usercorrelation function and the professional correlation function. Anysuitable feedback information which can be generated from the usercorrelation function and professional correlation function can bedisplayed here.

A first example of feedback information is the total error information644. This indicates the total finger positioning error for each sectionof the song. The total finger positioning error for each song section isdetermined from the total of the error values 706 (see FIG. 7) for eachfinger tip sensor 102A-D over the time period corresponding to thatsection. So, for example, if verse 1 occurs from elapsed time 0:20:00until elapsed time 0:41:59, then the error values 706 for each fingertip sensor during this time period are totaled. The total errors foreach finger tip sensor are then added together to give an overall value,which is then displayed (giving a total of 4 errors for verse 1 in thiscase). Advantageously, this information allows the user to see whichpart of their performance has the most and the least errors fingerpositioning errors, and thus which parts they need to improve.

An advice box 950 also gives tailored feedback to the user. Thisfeedback is generated on the basis of the user correlation function andprofessional correlation function comparison carried out by thecorrelation function engine 500. In this case, for example, it has beennoticed that the user has a particularly high finger positioning errorvalue for the “bridge” section of the song. The advice box 950 thussuggests that the user use the “song section repeat mode” (by selectingthe relevant section from the song section indicator 934) to practisethe “bridge” section. Any suitable feedback which can be generated bythe correlation function engine 500 from the user and professionalcorrelation function comparison may be displayed in the advice box 950.

A second example of feedback information is the force F_(Z) against timegraph 946 first illustrated in FIG. 8. Here, the graph is presented foreach finger tip sensor 102A-D individually, and the user may togglebetween the graphs for each finger tip sensor by touching the virtualtoggle buttons 948, 949 so as to see which of their fingers they need toimprove with regards to how much force is applied to the strings. Forconvenience, each finger is indicated by the colour of its correspondingfinger tip sensor. The total error (as given by equation (1)) could alsobe included for each graph (although this is not shown here). As analternative, the force F_(Z) against time graphs (user and professional)for each finger tip sensor 102A-D could be displayed on the same axes,with the graphs corresponding to each finger tip sensor beingdifferentiate by the finger tip sensor colour. Alternatively andadditionally, a graph of other characteristics may be shown, such as agraph of the differences in the audio captured by the microphone 404over time.

As well as the detailed feedback information 944, 946, a total score 952is also given for the user's performance. This may be calculated usingany suitable method, and can take into account any suitable range offactors. For example, contributing factors to the total score couldinclude the total number of finger position errors, the total forceerror for each finger, the speed at which the song was performed (withmore points being award for faster speeds), etc. Advantageously, byrecording a total score in this way, the user is able to obtain anoverall indicator of the quality of their performance.

FIG. 10 shows a process 1000 by which user correlation function data isobtained and compared with professional correlation data and by whichuser feedback is generated, according to embodiments.

The process starts at step 1002. At step 1004, a sample of usercorrelation function data is obtained. The user correlation functiondata includes image data ID from the camera 406 of the tablet 100, forcedata FD generated by the finger tip sensors 102A-D and/or audio data ADdetected by the microphone 404 of the tablet 100. At step 1004, the usercorrelation function data is processed so as to determine, for thesample, the positions of the finger tip sensors 102A-D with respect tothe strings 624 and frets 626 of the guitar and the average forceapplied by each of the finger tip sensors 102A-D.

The process then moves on to step 1006, where the processed usercorrelation function data is compared with the corresponding processedprofessional correlation function data. So, for example, the fingerpositioning of the user is compared with that of the professional andthe average force applied by each of the fingers of the user is comparedwith that of the professional.

At step 1008, following the comparison of the user and professionaldata, it is determined whether or not there are any errors in the user'sperformance which exceed a high error threshold. For the fingerpositioning, the high error threshold may simply be an error value of 2(so that an error in both the string and fret position, whichcorresponds to an error value of 2, is determined as exceeding the higherror threshold). For the force applied for each finger, then the higherror threshold may be an absolute force error (for example, 2 Newtons)or, alternatively, a percentage error (such as 15%).

In the case that there is an error which exceeds the appropriate higherror threshold, the process moves on to step 1010. Here, it isdetermined, for each error which has exceeded the high error threshold,whether this has occurred a predetermined number of consecutive times.This predetermined number is known as the error occurrence threshold.

In the case that the error occurrence threshold has been met for aparticular error, the process moves onto step 1012. In this case, areal-time advice message is provided to the user. This may be providedin the real-time advice box 938 on the performance screen 928. Thereal-time advice message provides tailored feedback to the user on theirperformance, taking into account the type of error that they have beenmaking.

For example, if a finger positioning error has consecutively occurred anumber of times equal to the error occurrence threshold, then an errortelling the user that they need to practise their finger positioning isdisplayed to the user. If the error appears to be being mainly caused bya single finger, then the colour of the finger tip sensor associatedwith that finger could also be included in the advice message.

Alternatively, if a force error has consecutively occurred a number oftimes equal to the error occurrence threshold, then an error telling theuser to apply more force to the strings (in the case that the user hasnot been applying enough force to the strings, for example) isdisplayed. Again, if the error appears to be being mainly caused by asingle finger, then the colour of the finger tip sensor associated withthat finger could also be included in the advice message.

Any suitable error occurrence threshold may be used. For example, theerror occurrence threshold may be 25, 50, 75 or 100 consecutive errors(corresponding to an elapsed track time of 0.25, 0.50, 0.75 and 1.00seconds, respectively, for a sample rate of 100 samples per second). Thedifferent types of error (that is, the finger positioning error and theforce error) may have the same error occurrence threshold or may havedifferent error occurrence thresholds.

In the case that the error occurrence threshold has not been met for anyof the detected errors, then no message is displayed to the user.Rather, the process moves onto step 1014 and the user correlationfunction data is stored as part of the user correlation function (forexample, as part of the table 600 in FIG. 6A).

Advantageously, by counting the number of consecutive errors made by theuser which exceed the high error threshold and by alerting the user whenthe number of consecutive errors is equal to the error occurrencethreshold, the user is made aware of significant errors which appear tobe being made consistently. At the same time, occasional errors (whichare unlikely to cause any real problems for the user) are ignored.

In the case that there is no error which is above the high errorthreshold at step 1008, the process moves onto step 1016. Here, it isdetermined whether or not there are any errors which are below a lowerror threshold. The low error threshold indicates an error below whichit is determined that the user is playing well.

For the finger positioning, the second threshold may simply be an errorvalue 706 of 1 (so that a perfectly correct finger position, whichcorresponds to an error value of 0, is determined as being below the lowerror threshold threshold). For the force applied for each finger, thelow error threshold may be an absolute force error (for example, 1Newton) or, alternatively, a percentage error (such as 5%).

In the case that there is an error which is below the appropriate lowerror threshold, the process moves on to step 1018. Here, it isdetermined, for each error which is below the low error threshold,whether this has occurred a predetermined number of consecutive times.This predetermined number may be the same error occurrence thresholdused in step 1010, or it may be different. In this embodiment, it is thesame error occurrence threshold that is used.

In the case that the error occurrence threshold has been met for aparticular error, then the process moves onto step 1020. In this case, areal-time encouragement message is provided to the user. This may beprovided in the real-time advice box 928. The real-time encouragementmessage provides tailored feedback to the user on their performance,taking into account the type of error which has been kept below the lowerror threshold.

For example, if a finger positioning error has been zero (below the lowerror threshold of 1, indicating perfectly correct finger positioning)for a consecutive number of times equal to the error occurrencethreshold, then a message telling the user that they have achievedperfect finger positioning is displayed. Alternatively, if a force errorfor a particular finger tip sensor has been below the low errorthreshold for a consecutive number of times equal to the erroroccurrence threshold, then a message telling the user that they haveachieved a very good level of force for that particular finger sensor isdisplayed.

In the case that the error occurrence threshold has not been met for anyof the detected low error values at step 1018, then no message isdisplayed to the user. Rather, the process moves onto step 1014 and theuser correlation function data is stored as part of the user correlationfunction.

Advantageously, by counting the number of consecutive low errors made bythe user and encouraging the user when the number of consecutive lowerrors is equal to the error occurrence threshold, the user is madeaware of the aspects of their musical performance for which they aredoing very well. At the same time, occasional low error values (whichmay occur randomly, even if the user is not playing very well overall)are ignored.

At step 1016, if it is determined that there are no errors which arebelow the low error threshold, then the process simply moves onto step1014, where the user correlation function data is stored as part of theuser correlation function.

From step 1014, the process moves onto step 1022, where it is determinedwhether or not it is time for the next sample to be obtained. So, forexample, if the sample rate is 100 samples per second (corresponding toa sample being taken every 0.01 seconds) with respect to the timeelapsed for the track, then it is determined whether or not 0.01 secondsof time elapsed for the track has passed since the last sample.

If the time for the next sample has not elapsed, then step 1022 isrepeated until the time for the next sample has elapsed. Once the timefor the next sample has elapsed, then the process then moves onto step1024, where it is determined whether or not the end of the song has beenreached. This is determined by comparing the total time elapsed for thetrack so far with the length of the track. If the song has not yetended, then the process goes back to step 1004, at which point the nextsample of user correlation function data is obtained. On the other hand,if the song has ended, then the process moves onto step 1026, where aperformance report is generated and displayed on the results screen 942.The process then ends at step 1028.

In embodiments, it is possible for several users, each with anindividual tablet 100 installed with the correlation function engine500, a guitar and a set of finger tip sensors to compete in onlinecompetitions so as to see which user is able to best replicate theperformance of the professional musician. A system 1100 by which thismay take place is illustrated in FIG. 11.

The system 1100 comprises a tablet 100A-C for each of the users A-C anda server 1102. Each of the tablets 100A-C is connected to the server bymeans of a network data connection 1104. This network data connection1104 may be a connection over the internet. In embodiments, the tablets100A-C may connect wirelessly to the internet via the wireless datatransceiver 410 comprised within each tablet.

According to embodiments, in order to take part in an onlinecompetition, a user may access an online competition screen 1200 viatheir tablet 100, as shown in FIG. 12. The online competition screen1200 lists all the online competitions 1202 which have been created andwhich the user may be able to join. Each online competition 1202 islisted with basic information. This includes the host name 1204 of theuser who created the competition, the artist 1206, track name 1208 anddifficulty 1210 of the song which has been chosen for the competition,and a start time 1212 for the competition. In embodiments, thecompetitions are listed in ascending order of start time (with thecompetition which is closest to starting being displayed first), and thestart time counts down in one second intervals until the start of thecompetition.

The user also has the option creating a new competition by pressing thevirtual button 1214. This is explained in more detail with respect toFIG. 13.

FIG. 13 shows a process 1300 by which a user may join a new competitionor may create a new competition for a song of their choice, according toembodiments.

The process starts at step 1302. At step 1304, the user logs in to thean online competition mode. By logging in, the user enters a usernameand password (for example) which is authenticated by the server 1102.Once the user is logged in, they are free to join or createcompetitions.

The process then moves to step 1306, where it is determined whether ornot a command has been received by the user to create a competition. Ifsuch a command has been received, then the process moves to step 1308.This marks the start of a competition creation process.

At step 1308, the user selects a song that is to be performed for thecompetition. This will be a song from the user's own song library. Theuser's song library is the collection of songs which have beenpreviously purchased by the user. At step 1310, the user then sets themaximum number of participants. For example, the user may set themaximum number of participants as 2, 3 or 4 participants. At step 1312,the user then sets the start time of the competition. This is the starttime 1212 that is displayed on the competition screen 1200 of thetablet. For example, the user may set the start time as 2, 3, 5 or 10minutes. Once step 1312 has been completed, the process moves onto step1314.

On the other hand, if no command is received by the tablet 100 to createa competition at step 1306, then it is determined at step 1316 as towhether or not a command has been received from the user to join anexiting competition. If a command has been received, then the processmoves on to step 1314. Alternatively, if a command has not beenreceived, then the process returns to step 1306, so as to once againcheck whether or not a command to create a competition has been issued.

At step 1314, a competition has either been joined by the user or hasbeen created by the user. Here, it is thus determined as to whether ornot the start time of the competition has elapsed (that is, has theincrementally decreasing start time reached zero). If the start time hasnot yet elapsed, then the check at step 1314 continually takes placeuntil the start time has elapsed. Once the start time has elapsed, thenthe process moves onto step 1318.

At step 1318, the competition takes place. Here, correlation functiondata is taken for the performance of each of the users so as to create acorrelation function for each user. In this case, a screen similar tothe performance screen 928 is displayed to each of the users. However,since this is a competition, certain features of the performance screenwill be disabled. In particular, the speed bar 936, progress bar 932 andsong section indicator 934 functionality will be disabled, so that nouser can adjust the speed of the song or skip the song (these parametersmay, however, be chosen by the host of the competition as steps inaddition to steps 1308, 1310 and 1312 when the competition is created).

Once the song has been completed, the process moves onto step 1320,where the correlation function for each of the users is analysed so asto create a total score for each user (similar to the total score 952displayed on the results screen 942 in the previously described singleplayer mode). The users are then ranked according to their total scores.

The process then moves onto step 1322, where the rankings are displayedto the users and the total score of each user is recorded to theirrespective online competition profile. By recording the total score ofeach user to their online competition profile, this information can beshared with other users for future competitions. So, for example,experienced users may be able to seek competitions hosted by otherexperienced users (who will have relatively high total score records),where as less experienced users may be able to seek competitions hostedby other less experienced users (who will have relatively low totalscore records). Advantageously, users can thus participate incompetitions with other users with similar levels of ability.

It will be appreciated that although embodiments of the disclosure havebeen described as being implemented via software on a tablet computer100, it will be appreciated that other suitable implementation methodsof the disclosure are envisaged. For example, embodiments of thedisclosure could be implemented in software on a mobile phone (such as asmartphone), a laptop or desktop computer, a smart TV, etc. In fact,embodiments of the disclosure may be implemented in any device which canrun a correlation function engine 500 in software or hardware and whichis able to generate sufficiently detailed image data ID and/or audiodata AD and to receive force data FD from the finger tip sensors 102A-D.

Although described with reference to a tablet type computer, thedisclosure should not be construed as being limited to this. Forexample, the user may wear a head mountable display such as glasses thatinclude a heads-up display.

Obviously, numerous modifications and variations of the presentdisclosure are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, thedisclosure may be practiced otherwise than as specifically describedherein.

In so far as embodiments of the disclosure have been described as beingimplemented, at least in part, by software-controlled data processingapparatus, it will be appreciated that a non-transitory machine-readablemedium carrying such software, such as an optical disk, a magnetic disk,semiconductor memory or the like, is also considered to represent anembodiment of the present disclosure.

CLAUSES

Features of the disclosure are provided in the following numberedparagraphs.

-   1. A method of providing feedback to a user of a string instrument,    comprising the steps:    -   detecting the force of a user's finger on at least one string at        a given time;    -   detecting the position of the user's finger on the at least one        string at the given time;    -   comparing the detected force of the user's finger and the        position of the user's finger on the at least one string at the        given time with a predetermined value of force and position at        the given time; and    -   generating feedback to display to the user on the basis of the        said comparison.-   2. A method according to clause 1 further comprising detecting the    frequency of vibration on the user's finger at the given time,    calculating the frequency spectrum of the vibration at the given    time and determining the number of strings played by the user at the    given time.-   3. A method according to clause 1 or 2 further comprising receiving    an image of the string instrument and determining the position of    the user's finger on the string instrument from the received image.-   4. A method according to clause 3, wherein the position of the    user's finger is determined by performing edge detection on the    received image.-   5. A method according to clause 3 or 4, comprising performing image    detection on the received image, wherein the position of the user's    finger is determined by detecting a coloured finger sensor in the    received image.-   6. A method according to clause 1 to 5, comprising determining the    difference between the detected force and the predetermined value of    force at the given time and determining an error based on the    determined differences over a predetermined time.-   7. A method according to clause 1 to 6, comprising determining the    difference between the detected force and the predetermined value of    force at the given time, and determining when the difference exceeds    a threshold over a predetermined period of time.-   8. A device for providing feedback to a user of a string instrument,    comprising:    -   receiving circuitry operable in use to communicate with at least        one finger sensor capable of detecting the force of a finger on        a string, a camera operable in use to capture an image and a        display operable in use to display data, wherein the receiving        circuitry, camera and display are connected to processing        circuitry which is operable to:    -   detect, from the receiving circuitry, the force of a user's        finger on at least one string at a given time;    -   detect, from the camera, the position of the user's finger on        the at least one string at the given time;    -   compare the detected force of the user's finger and the position        of the user's finger on the at least one string at the given        time with a predetermined value of force and position at the        given time; and    -   generate, on the basis of the said comparison, feedback to        display on the display.-   9. A device according to clause 8 wherein the processing circuitry    is further operable to detect, from the receiving circuitry, the    frequency of vibration on the user's finger at the given time, and    to calculate the frequency spectrum of the vibration at the given    time and determining the number of strings played by the user at the    given time.-   10. A device according to clause 8 or 9 wherein the processing    circuitry is further operable in use to receive an image of the    string instrument from the camera, and from the received image is    operable to determine the position of the user's finger on the    string instrument.-   11. A device according to clause 10, wherein the position of the    user's finger is determined by performing edge detection on the    received image.-   12. A device according to clause 10 or 11, wherein the processing    circuitry is further operable in use to perform image detection on    the received image, wherein the position of the user's finger is    determined by detecting a coloured finger sensor in the received    image.-   13. A device according to clause 8 to 12, wherein the processing    circuitry is further operable in use to determine the difference    between the detected force and the predetermined value of force at    the given time and to determine an error based on the determined    differences over a predetermined time.-   14. A device according to clause 8 to 13, wherein the processing    circuitry is further operable in use to determine the difference    between the detected force and the predetermined value of force at    the given time, and determine when the difference exceeds a    threshold over a predetermined period of time.-   15. A tablet device for providing feedback to a user of a string    instrument, comprising:    -   receiving circuitry operable in use to communicate with at least        one finger sensor capable of detecting the force of a finger on        a string, a camera operable in use to capture an image and a        display operable in use to display data, wherein the receiving        circuitry, camera and display are connected to processing        circuitry which is operable to:    -   detect, from the receiving circuitry, the force of a user's        finger on at least one string at a given time;    -   detect, from the camera, the position of the user's finger on        the at least one string at the given time;    -   compare the detected force of the user's finger and the position        of the user's finger on the at least one string at the given        time with a predetermined value of force and position at the        given time; and    -   generate, on the basis of the said comparison, feedback to        display on the display.-   16. Computer software comprising computer readable code which when    loaded onto a computer configures the computer to perform a method    according to clause 1 to 7.-   17. A storage medium configured to store the computer software of    claim 16 thereon or therein.-   18. A method, device, tablet device or software as substantially    hereinbefore described with reference to the accompanying drawings.

The invention claimed is:
 1. A method of providing feedback to a user ofa string instrument, comprising: detecting a force of a user's finger onat least one string at a given time; detecting a position of the user'sfinger on the at least one string at the given time; comparing thedetected force of the user's finger and the position of the user'sfinger on the at least one string at the given time with at least twopredetermined values of force and position at the given time; andgenerating feedback to display to the user on the basis of the saidcomparison.
 2. The method according to claim 1, further comprising:detecting a frequency of vibration on the user's finger at the giventime; calculating a frequency spectrum of the vibration at the giventime; and determining a number of strings played by the user at thegiven time.
 3. The method according to claim 1, further comprising:receiving an image of the string instrument and determining the positionof the user's finger on the string instrument from the received image.4. The method according to claim 3, wherein the position of the user'sfinger is determined by performing edge detection on the received image.5. The method according to claim 3, further comprising: performing imagedetection on the received image, wherein the position of the user'sfinger is determined by detecting a colored finger sensor in thereceived image.
 6. The method according to claim 1, further comprising:determining a difference between the detected force and at least one ofthe predetermined values of force at the given time; and determining anerror based on the determined differences over a predetermined time. 7.The method according to claim 1, further comprising: determining adifference between the detected force and at least one of thepredetermined values of force at the given time; and determining whenthe difference exceeds a threshold over a predetermined period of time.8. A device for providing feedback to a user of a string instrument,comprising: receiving circuitry configured to communicate with at leastone finger sensor capable of detecting a force of a finger on a string,a camera configured to capture an image and a display configured todisplay data, wherein the receiving circuitry, camera and display areconnected to processing circuitry which is configured to: detect, fromthe receiving circuitry, the force of a user's finger on at least onestring at a given time; detect, from the camera, a position of theuser's finger on the at least one string at the given time; compare thedetected force of the user's finger and the position of the user'sfinger on the at least one string at the given time with at least twopredetermined values of force and position at the given time; andgenerate, on the basis of the said comparison, feedback to display onthe display.
 9. The device according to claim 8, wherein the processingcircuitry is further configured to detect, from the receiving circuitry,a frequency of vibration on the user's finger at the given time;calculate a frequency spectrum of the vibration at the given time; anddetermine a number of strings played by the user at the given time. 10.The device according to claim 8, wherein the processing circuitry isfurther configured to receive an image of the string instrument from thecamera; and determine the position of the user's finger on the stringinstrument from the received image.
 11. The device according to claim10, wherein the position of the user's finger is determined byperforming edge detection on the received image.
 12. The deviceaccording to claim 10, wherein the processing circuitry is furtherconfigured to perform image detection on the received image, and theposition of the user's finger is determined by detecting a coloredfinger sensor in the received image.
 13. The device according to claim8, wherein the processing circuitry is further configured to determine adifference between the detected force and at least one of thepredetermined values of force at the given time; and determine an errorbased on the determined differences over a predetermined time.
 14. Thedevice according to claim 8, wherein the processing circuitry is furtherconfigured to determine a difference between the detected force and atleast one of the predetermined values of force at the given time; anddetermine when the difference exceeds a threshold over a predeterminedperiod of time.
 15. A tablet device for providing feedback to a user ofa string instrument, comprising: receiving circuitry configured tocommunicate with at least one finger sensor capable of detecting a forceof a finger on a string, a camera configured to operable in use tocapture an image and a display configured to display data, wherein thereceiving circuitry, camera and display are connected to processingcircuitry which is configured to: detect, from the receiving circuitry,the force of a user's finger on at least one string at a given time;detect, from the camera, a position of the user's finger on the at leastone string at the given time; compare the detected force of the user'sfinger and the position of the user's finger on the at least one stringat the given time with at least two predetermined values of force andposition at the given time; and generate, on the basis of the saidcomparison, feedback to display on the display.
 16. Computer softwarecomprising computer readable code which when loaded onto a computerconfigures the computer to perform the method according to claim
 1. 17.A non-transitory storage medium configured to store the computersoftware of claim 16 thereon or therein.